[stormshield] Add ECS event categorization and network mappings by arenard · Pull Request #20031 · elastic/integrations

arenard · 2026-07-08T10:07:04Z

Proposed commit message

Set event.kind, event.category and event.type for every SNS log family through a table-driven Painless script keyed on logtype, with allowed/denied event.type modifiers derived case-insensitively from the rule action (pass/block; an empty action corresponds to a rule set to Log and adds no modifier). The periodic statistics families (monitor, filterstat, count, routerstat, authstat, ipsecstat) are tagged event.kind: metric so health telemetry can be excluded from detection scopes with a single filter. Unknown families fall back to event/network/info, covered by a dedicated test sample. The script only runs on documents without an error, so the pipeline_error kind set by sub-pipeline failure handlers is preserved.

Add the ECS network mappings tracked in #10114: rcvd to destination.bytes (mirroring the existing sent mapping), network.bytes as the sum of both, srcif/srcifname/dstif/dstifname copied to observer.ingress/observer.egress interface fields, and network.direction computed with private ranges as internal networks. Promote alarm fields (alarmid to event.code, risk to event.risk_score, pri to event.severity on the alarm scale where 1 is major and 4 is minor), HTTP proxy fields (arg to url.original, url.domain from destination.domain), and populate related.hosts. Guard the ASN geoip processors with null checks, matching the geo processors.

Fix the count sub-pipeline, which failed on every count log since the package was created: the script called splitByToken, which does not exist in Painless (runtime error: dynamic method [java.lang.String, splitByToken/2] not found), and collapsed all rules into a single map. Rule counters now form a sorted array under stormshield.metadata.rule_stats, and a counter that does not parse (non-numeric value, out-of-range category or unexpected key shape) is skipped instead of sending the whole document to pipeline_error.

Add pipeline test samples for the alarm, auth, web, vpn, xvpn, count and filter families plus an unknown-family sample, an Alarms (IPS) dashboard keyed on the new categorization fields, and the categorization table in the README.

Design notes

All new mappings are copies: no stormshield.* field is renamed, moved or removed, and event.action keeps the verbatim vendor value, so existing dashboards and saved queries are unaffected. This keeps the change a minor version bump.
The single log data stream is kept, following the fortinet_fortigate / cisco_asa / sonicwall_firewall precedent for multi-family syslog firewall packages; the categorization script follows the sonicwall_firewall table-driven pattern.
event.outcome derivation and deeper per-family enrichment (vpn IKE fields, sandboxing verdicts, the v5 restapi family) are planned as follow-ups.
Test samples follow the log format examples from the official Stormshield audit log documentation, with addresses in RFC 1918/5737 ranges; they exercise uppercase (Block), lowercase (pass) and empty action values, and the unknown-family fallback.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

elastic-package check, test pipeline, test static and test system (udp, tcp, tls) all pass against stack 9.4.3
The dashboard imports cleanly into Kibana 9.4.3, every panel is scoped by the dashboard-level data_stream.dataset and stormshield.logtype filters, and every aggregated field is aggregatable and declared in the package field files
sample_event.json regenerated through elastic-package test system --generate
Fields referenced by the existing Overview dashboard (event.action, log.syslog.severity.name, stormshield.fw, stormshield.logtype, stormshield.in_bytes, stormshield.out_bytes) are untouched

How to test this PR locally

cd packages/stormshield
elastic-package check
elastic-package test pipeline
elastic-package stack up -d
elastic-package test system

Related issues

Relates [stormshield] Follow-up tasks for new integration #10114
Requires [stormshield] Fix pipeline descriptions and correct documentation claims #20030 (the branch is stacked on it; will be rebased once it merges)

cla-checker-service · 2026-07-08T10:07:10Z

💚 CLA has been signed

github-actions · 2026-07-08T10:08:31Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

elastic-vault-github-plugin-prod · 2026-07-08T10:31:38Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Fix the copy-pasted descriptions of the default and count ingest pipelines, remove field definitions that never appear in events (the pipeline consumes tz into event.timezone, converts duration into event.duration, renames msg to message, and drops origdst/origdstport after converting them to destination fields), and correct the documentation: the agent syslog processor is configured for RFC5424, so the README no longer presents the Legacy format as supported, and the compatibility section now covers SNS 4.x and 5.x, which share the same WELF key-value audit log format. Also document how to collect SNS IPFIX flow records with the NetFlow integration, update vendor links to the SNS v5 documentation, and use arrows for menu paths per the documentation style guide.

infra-vault-gh-plugin-prod · 2026-07-10T16:01:38Z

Pinging @elastic/integration-experience (Team:Integration-Experience)

vera-review-bot · 2026-07-10T16:17:40Z

+      params:
+        logtypes:
+          alarm:      { kind: alert,  category: [intrusion_detection, network], type: [info] }
+          auth:       { kind: event,  category: [authentication], type: [start] }


Severity: 🟡 Medium confidence: medium path: packages/stormshield/data_stream/log/elasticsearch/ingest_pipeline/default.yml:661

Authentication events never set event.outcome, so successful and failed logins are indistinguishable by outcome; derive event.outcome from the auth error field.

Details

The new categorization maps the auth family to event.kind: event, event.category: [authentication], event.type: [start] for every auth record regardless of result. The pipeline never sets event.outcome, so a successful login (error=0, message "User authenticated") and a failed login (error=1, message "Authentication failed") produce identical categorization — this is visible in the new test fixture, where both auth events emit event.type: [start] and no event.outcome. For the authentication category, event.outcome (success/failure) is the field detection rules and dashboards use to isolate failed logins, and the raw error flag (0 = success, 1 = failure) is already parsed into stormshield.metadata.error. Without it, an auth-failure panel or rule cannot be built on this data.

Recommendation:

Set event.outcome for the auth family from the error flag before it is moved under stormshield.metadata. For example:

- set: tag: set_event_outcome_auth_success field: event.outcome value: success if: ctx.stormshield?.logtype == 'auth' && ctx.stormshield?.error == '0' - set: tag: set_event_outcome_auth_failure field: event.outcome value: failure if: ctx.stormshield?.logtype == 'auth' && ctx.stormshield?.error != null && ctx.stormshield.error != '0'

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Good catch, this is implemented in the follow-up PR #20085 (stacked on this one), which derives event.outcome from the vendor error return code on the auth, server and xvpn families (the exact families the vendor documentation lists for that field) with pipeline test coverage for both outcomes.

vera-review-bot · 2026-07-10T16:17:40Z

+  changes:
+    - description: Fix ingest pipeline descriptions and remove field definitions that never appear in events (`stormshield.tz`, `stormshield.duration`, `stormshield.msg`, `stormshield.origdst`, `stormshield.origdstport`).
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/20030


Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:23

Changelog links a different PR number

Details

This changelog entry's link: points at pull/20030, but it was added in PR #20031. It is likely a leftover template placeholder or a copy from another PR.

Recommendation:

Point the changelog link at this PR:

link: https://github.com/elastic/integrations/pull/20031

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

These two entries belong to the 1.5.3 release introduced by #20030; they only appear in this diff because the branch is stacked on that PR. The links are correct, this resolves once #20030 merges and this branch is rebased.

vera-review-bot · 2026-07-10T16:17:40Z

+      link: https://github.com/elastic/integrations/pull/20030
+    - description: Correct the syslog format and SNS version compatibility statements in the documentation, and document IPFIX flow collection through the NetFlow integration.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/20030


Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:26

Changelog links a different PR number

Details

This changelog entry's link: points at pull/20030, but it was added in PR #20031. It is likely a leftover template placeholder or a copy from another PR.

Recommendation:

Point the changelog link at this PR:

link: https://github.com/elastic/integrations/pull/20031

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Same as above: this 1.5.3 entry comes from #20030, on which this branch is stacked. The link is correct and this resolves after #20030 merges and the branch is rebased.

Set event.kind, event.category and event.type for every SNS log family through a table-driven Painless script keyed on logtype, with allowed/denied event.type modifiers derived case-insensitively from the rule action. The periodic statistics families (monitor, filterstat, count, routerstat, authstat, ipsecstat) are tagged event.kind: metric so health telemetry can be excluded from detection scopes with a single filter. Unknown families fall back to event/network/info, covered by a dedicated test sample. The script only runs on documents without an error, so the pipeline_error kind set by sub-pipeline failure handlers is preserved. Add the ECS network mappings tracked in elastic#10114: rcvd to destination.bytes (mirroring the existing sent mapping), network.bytes as the sum of both, srcif/srcifname/dstif/dstifname copied to observer.ingress and observer.egress interface fields, and network.direction computed with private ranges as internal networks. Promote alarm fields (alarmid to event.code, risk to event.risk_score, pri to event.severity on the alarm scale where 1 is major and 4 is minor), HTTP proxy fields (arg to url.original, url.domain from destination.domain), and populate related.hosts. All mappings are copies; no stormshield.* field moves or changes location. Guard the ASN geoip processors with null checks, matching the geo processors. Fix the count sub-pipeline, which failed on every count log since creation: the script called splitByToken, which does not exist in Painless, and collapsed all rules into a single map. Rule counters are now a sorted array under stormshield.metadata.rule_stats. Extend the metadata move list with the documented alarm, proxy, vpn and auth fields so they land under stormshield.metadata instead of being left unmapped. Add pipeline test samples for the alarm, auth, web, vpn, xvpn, count and filter families, derived from the log format examples in the official Stormshield documentation with sanitized addresses, including uppercase and empty action values. Add an Alarms (IPS) dashboard keyed on the new categorization fields. Document the categorization table in the README.

elastic-vault-github-plugin-prod · 2026-07-10T21:21:33Z

Changelog link mismatch — expected https://github.com/elastic/integrations/pull/20031 in the following file(s):

packages/stormshield/changelog.yml

Tip

If expected, add the changelog-link-check:skip label to skip this check. Or, if an issue link was intended, use .../issues/<n> instead.

View Buildkite build
@arenard

infra-vault-gh-plugin-prod · 2026-07-10T21:21:36Z

💔 Build Failed

Buildkite Build
Commit: 4ce4236

Failed CI Steps

📜 Check changelog PR links

History

💔 Build #45871 failed a1623ea
💔 Build #45869 failed 069f288
💔 Build #45822 failed d5a60c4
💔 Build #45794 failed 7a141fe
💔 Build #45791 failed 5f9b258

vera-review-bot · 2026-07-10T21:54:46Z

+  changes:
+    - description: Fix ingest pipeline descriptions and remove field definitions that never appear in events (`stormshield.tz`, `stormshield.duration`, `stormshield.msg`, `stormshield.origdst`, `stormshield.origdstport`).
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/20030


Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:23

Changelog links a different PR number

Details

This changelog entry's link: points at pull/20030, but it was added in PR #20031. It is likely a leftover template placeholder or a copy from another PR.

Recommendation:

Point the changelog link at this PR:

link: https://github.com/elastic/integrations/pull/20031

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

vera-review-bot · 2026-07-10T21:54:46Z

+      link: https://github.com/elastic/integrations/pull/20030
+    - description: Correct the syslog format and SNS version compatibility statements in the documentation, and document IPFIX flow collection through the NetFlow integration.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/20030


Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:26

Changelog links a different PR number

Details

This changelog entry's link: points at pull/20030, but it was added in PR #20031. It is likely a leftover template placeholder or a copy from another PR.

Recommendation:

Point the changelog link at this PR:

link: https://github.com/elastic/integrations/pull/20031

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

vera-review-bot · 2026-07-10T21:54:47Z

Review summary

Issues found across the latest commits 4ce4236 — 2 medium

🟡 Changelog links a different PR number (link) (Unresolved)
🟡 Changelog links a different PR number (link) (Unresolved)

Issues found across earlier commits a1623ea — 3 medium

🟡 Authentication events never set event.outcome, so successful and failed logins are indistinguishable by outcome (link) (Unresolved)
🟡 Changelog links a different PR number (link) (Unresolved)
🟡 Changelog links a different PR number (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

arenard force-pushed the stormshield-ecs-categorization branch 2 times, most recently from a0275dc to 59232a6 Compare July 8, 2026 15:48

andrewkroh removed the needs CLA User must sign the Elastic Contributor License before review. label Jul 8, 2026

arenard force-pushed the stormshield-ecs-categorization branch 5 times, most recently from 11084e6 to d5a60c4 Compare July 9, 2026 15:33

arenard changed the title ~~stormshield: add ECS event categorization and network mappings~~ [stormshield] Add ECS event categorization and network mappings Jul 9, 2026

arenard force-pushed the stormshield-ecs-categorization branch from d5a60c4 to 069f288 Compare July 10, 2026 11:24

arenard marked this pull request as ready for review July 10, 2026 11:24

arenard requested a review from a team as a code owner July 10, 2026 11:24

arenard force-pushed the stormshield-ecs-categorization branch from 069f288 to a1623ea Compare July 10, 2026 11:29

arenard marked this pull request as draft July 10, 2026 11:37

arenard marked this pull request as ready for review July 10, 2026 12:10

arenard mentioned this pull request Jul 10, 2026

[stormshield] Add outcome, identity and HTTP mappings and extend the metadata allow-list #20085

Open

7 tasks

andrewkroh added the Team:Integration-Experience Security Integrations Integration Experience [elastic/integration-experience] label Jul 10, 2026

vera-review-bot Bot reviewed Jul 10, 2026

View reviewed changes

arenard mentioned this pull request Jul 10, 2026

[stormshield] Add thematic dashboards and rework the Overview #20088

Draft

9 tasks

arenard force-pushed the stormshield-ecs-categorization branch from a1623ea to 4ce4236 Compare July 10, 2026 21:17

vera-review-bot Bot reviewed Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

arenard commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Design notes

Checklist

Author's Checklist

How to test this PR locally

Related issues

Uh oh!

cla-checker-service Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Elastic Docs Style Checker (Vale)

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 8, 2026

🚀 Benchmarks report

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 10, 2026

Uh oh!

vera-review-bot Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

arenard Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

arenard Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

arenard Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 10, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

vera-review-bot Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot commented Jul 10, 2026

Review summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

arenard commented Jul 8, 2026 •

edited

Loading

cla-checker-service Bot commented Jul 8, 2026 •

edited

Loading

github-actions Bot commented Jul 8, 2026 •

edited

Loading

arenard Jul 10, 2026 •

edited

Loading

arenard Jul 10, 2026 •

edited

Loading

infra-vault-gh-plugin-prod Bot commented Jul 10, 2026 •

edited

Loading